Method and apparatus for the projection of images, video, and/or holograms generated by a computer simulation

ABSTRACT

A method includes running a computer simulation that is controlled at least in part by a handheld controller, displaying images generated by the computer simulation on a display, and projecting at least one image generated by the computer simulation from an image projector coupled to the handheld controller. A system includes a display, a handheld controller, an image projector coupled to the handheld controller, and a processor based apparatus. Another system includes a display, a handheld controller, an image projector that is housed in a unit configured to be releasably docked to the handheld controller, and a processor based apparatus. Storage mediums storing one or more computer programs are also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/994,027, filed on Jan. 12, 2016, entitled “METHOD AND APPARATUS FORTHE PROJECTION OF IMAGES, VIDEO, AND/OR HOLOGRAMS GENERATED BY ACOMPUTER SIMULATION,” which claims the benefit of U.S. ProvisionalPatent Application No. 62/234,570, filed on Sep. 29, 2015, entitled“METHOD AND APPARATUS FOR THE PROJECTION OF IMAGES, VIDEO, AND/ORHOLOGRAMS GENERATED BY A COMPUTER SIMULATION,” the entire contents anddisclosures of which are both hereby fully incorporated by referenceherein in their entireties.

BACKGROUND OF THE INVENTION 1. Field of the Invention

Embodiments of the present invention relate generally to computersimulation output technology, and more specifically to image, video, andaudio output technology that may be employed by computer simulations,such as computer games and video games.

2. Discussion of the Related Art

Computer games, such as video games, have become a popular source ofentertainment. Computer games are typically implemented in computer gamesoftware applications and are often run on game consoles, entertainmentsystems, desktop, laptop, and notebook computers, portable devices,pad-like devices, etc. Computer games are one type of computersimulation.

The user of a computer game is typically able to view the game play on adisplay and control various aspects of the game with a handheldcontroller, such as a game controller, game pad, wand, motion sensingcontroller, joystick, mouse, or other types of handheld controllers.Computer games typically also include audio output so that the user canhear sounds generated by the game, such as for example, the soundsgenerated by other players' characters like voices, footsteps, physicalconfrontations, gun shots, explosions, car chases, car crashes, etc.

SUMMARY OF THE INVENTION

One embodiment provides a method, comprising: running, with a processorbased apparatus, a computer simulation that is controlled at least inpart by a handheld controller; displaying images generated by thecomputer simulation on a display; and projecting at least one imagegenerated by the computer simulation from an image projector coupled tothe handheld controller.

Another embodiment provides a system, comprising: a display; a handheldcontroller; an image projector coupled to the handheld controller; and aprocessor based apparatus in communication with the display, thehandheld controller, and the image projector; wherein the processorbased apparatus is configured to execute steps comprising: running acomputer simulation that is controlled at least in part by the handheldcontroller; sending images generated by the computer simulation to thedisplay; and sending at least one image generated by the computersimulation to the image projector.

Another embodiment provides a non-transitory computer readable storagemedium storing one or more computer programs configured to cause aprocessor based system to execute steps comprising: running a computersimulation that is controlled at least in part by a handheld controller;sending images generated by the computer simulation to a display; andsending at least one image generated by the computer simulation to animage projector coupled to the handheld controller.

Another embodiment provides a non-transitory computer readable storagemedium storing one or more computer programs configured to cause aprocessor based system to execute steps comprising: running a computersimulation that is controlled at least in part by a handheld controller;sending images generated by the computer simulation to a display; andsending at least one image generated by the computer simulation to animage projector that is housed in a unit configured to be releasablydocked to the handheld controller.

A better understanding of the features and advantages of variousembodiments of the present invention will be obtained by reference tothe following detailed description and accompanying drawings which setforth an illustrative embodiment in which principles of embodiments ofthe invention are utilized.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of embodiments ofthe present invention will be more apparent from the following moreparticular description thereof, presented in conjunction with thefollowing drawings wherein:

FIGS. 1, 2, and 3 are block diagrams illustrating a system in accordancewith embodiments of the present invention;

FIGS. 4A, 4B, and 4C are block diagrams illustrating example uses of animage projector in accordance with embodiments of the present invention;

FIGS. 5A, 5B, and 5C are block diagrams illustrating the adjustment ofan angle of projection of an image projector in accordance withembodiments of the present invention;

FIGS. 6A and 6B are block diagrams illustrating the use of a panel andan image projector to form a hologram in accordance with embodiments ofthe present invention;

FIG. 7 is a block diagram illustrating the use of fog, or other tinyparticles suspended in air, and an image projector to form a hologram inaccordance with embodiments of the present invention;

FIG. 8 is a block diagram illustrating example components of a unit inaccordance with embodiments of the present invention;

FIG. 9 is a block diagram illustrating example components of a handheldcontroller in accordance with embodiments of the present invention;

FIG. 10 is a flow diagram illustrating a method in accordance withembodiments of the present invention; and

FIG. 11 is a block diagram illustrating a processor basedapparatus/system that may be used to run, implement, and/or execute anyof the methods, schemes, and techniques shown and described herein inaccordance with the embodiments of the present invention.

DETAILED DESCRIPTION

As mentioned above, computer games, such as video games, have become apopular source of entertainment. Currently, the video output of mostcomputer games is typically displayed on a single display, such as forexample a monitor or television (TV). As such, when playing a computergame the user typically just keeps looking at the same spot, namely thesingle display.

In some embodiments, the present invention makes computer games andother types of computer simulations more interactive and entertaining byadding an image projection output that is used in addition to thetraditional video display output. In some embodiments, images areprojected from the handheld controller that is used by the user tocontrol the computer simulation. This allows the user to display theextra projected images on a wall, the floor, the ceiling, or anywherethe user points the handheld controller. For example, the user canproject a map on the floor or on a wall in order to explore the town,forest, etc., in the game.

In some embodiments, holographic images are projected. That is, imagesprojected from the handheld controller may be used to form a hologram.This allows, for example, a person's face or entire body, such as acharacter in a game, to be projected into the air and suspended there.And in some embodiments, the system also generates audio that isassociated with the projected images. This allows, for example, a voiceto be added to the hologram of the game character, which in turn allowsthe hologram to speak to the user to, for example, provide orders tofollow the next mission or some other instructions or information.

Referring to FIG. 1, there is illustrated a block diagram of a system100 in accordance with an embodiment of the present invention. Thesystem 100 includes a processor based apparatus 102, such as a gameconsole or entertainment system. The processor based apparatus 102 iscapable of running a computer simulation, such as a computer game orvideo game. The computer simulation is controlled at least in part by ahandheld controller 104, such as a game controller, game pad, wand, orany other type of handheld controller.

A display 106 is used for displaying images generated by the computersimulation. The display 106 may comprise any type of display, such as amonitor, TV, or any other type of display. The display 106 may becoupled to, or in communication with, the processor based apparatus 102via a wired or wireless connection.

In some embodiments, an image projector 108 is coupled to the handheldcontroller 104. An image projector is an optical device that projectsimages onto a surface. A small scale or miniaturized image projector isknown as a pico projector, handheld projector, or mobile projector. Somemajor imager technologies employed by pico projectors include DigitalLight Processing (DLP), Liquid Crystal on Silicon (LCoS), and laser. Theimage projectors discussed herein may comprise any of these types ofimage projectors, as well as any other types of image projectors.

In some embodiments, the image projector 108 is coupled to the handheldcontroller 104 by being built into the handheld controller 104. Forexample, the image projector 108 may be included and/or mounted in thehousing of the handheld controller 104. Or, the image projector 108 maybe integrated, incorporated, and/or combined with the handheldcontroller 104 in some other way. A version of this type of coupling ofthe image projector 108 to the handheld controller 104 will be discussedbelow, with an example provided in FIG. 9.

In some embodiments, the image projector 108 is coupled to the handheldcontroller 104 by being housed in a unit 110 that is capable of beingremovably or releasably attached or docked to the handheld controller104, which is what is illustrated in FIG. 1. For example, the unit 110and the handheld controller 104 may be configured such that the unit 110can be docked with the handheld controller 104, and then easilyundocked. The unit 110 may be small so that it can be docked or attachedto the handheld controller 104 using a mount, holder, bracket, orsimilar releasable attachment or docking mechanism. In this way, in someembodiments, the image projector 108 is housed in the unit 110 that isconfigured to be releasably docked to the handheld controller 104. Insome embodiments, the unit 110 is operational whether or not it isdocked with the handheld controller 104. For example, a stand-alone useof the unit 110 will be discussed below.

During operation of the system 100, the processor based apparatus 102runs a computer simulation that is controlled at least in part by thehandheld controller 104, which is manipulated by a user. Imagesgenerated by the computer simulation are sent to the display 106 wherethe images are display to the user. In addition, at least one imagegenerated by the computer simulation is sent to the image projector 108.

In some embodiments, the at least one image generated by the computersimulation is sent to the image projector 108 via wireless streaming.For example, a WiFi or other wireless connection may be used by theprocessor based apparatus 102 to send the at least one image generatedby the computer simulation to the image projector 108.

Once received, the image projector 108 projects the at least one imagegenerated by the computer simulation. The projection of the image by theimage projector 108 is indicated by the beam 109. In the example shownin FIG. 1, the image projector 108 projects the image onto the floor,which is indicated by the image 112. It should be understood, however,that the image 112 can be projected onto any surface, such as a wall orceiling, or even into the air to form a hologram, as will be discussedbelow.

In some embodiments, the unit 110 also includes one or more audiospeakers, such as stereo speakers, or an audio output jack (forconnection to external speakers) so that sounds can be heard by the userwhile viewing the projected image 112. For example, the computersimulation may generate one or more audio signals that are associatedwith the at least one image generated by the computer simulation that issent to the image projector 108. The processor based apparatus 102 thenprovides the one or more audio signals to the unit 110 or wherever theaudio circuity associated with the projector 108 is located. In someembodiments, the processor based apparatus 102 provides the one or moreaudio signals to the unit 110 via wireless streaming, such as by theabove-mentioned WiFi or other wireless connection. Thus, as is the usualuse case, the handheld controller 104 is generally connected to theprocessor based apparatus 102, such as a game console, via a wirelessconnection for the gaming control (or more generally, for simulationcontrol). The unit 110 may be connected through another wireless linkand stream audio and video (AN) using some compressed data format.

Thus, the system 100 provides a solution that gives the user a secondaryinteractive image 112 near or around user. By moving and/or pointing thehandheld controller 104, the user can project the image 112 wherever heor she wants, such as on the floor, or on the wall, or on a table, etc.This provides a more entertaining and exciting experience for the userthan previous systems in which the user played a game by viewing asingle display, such as a single PC monitor or a TV. In such previoussystems there was no additional vision involved during the game play,and that disadvantage is solved by the system 100. The system 100provides the user with a secondary, interactive image/screen in areasonably compact system. Furthermore, the system 100 provides acompact video/audio system that can be integrated with, or into, anexisting handheld controller using projecting technology.

FIG. 2 illustrates an example of how the system 100 can be used toproject an image on a wall. Specifically, the system 100 is againillustrated, but this time the handheld controller 104 and attached unit110 are moved to a different position. In the illustrated position theimage projector 108 projects the image 112 onto a side wall 120. Theprojection of the image by the image projector 108 is indicated by thebeam 109.

It was mentioned above that in some embodiments, the unit 110 isoperational whether or not it is docked with the handheld controller104, and that an example of an undocked, stand-alone use of the unit 110would be discussed. FIG. 3 illustrates a stand-alone use of the unit 110in accordance with an embodiment of the present invention. Specifically,the system 100 is again illustrated, but this time the unit 110 isundocked or detached from the handheld controller 104 and moved to adifferent location. The unit 110 is still fully operational, and as suchthe image projector 108 projects the image 112 onto whatever surface theprojector 108 is pointed at, such as the floor, a wall, the ceiling,etc. In the illustrated example, the image projector 108 projects theimage 112 onto the back wall 122. Again, the projection of the image bythe image projector 108 is indicated by the beam 109. By having the unit110 remain in one location/position and projecting on the back wall 122next to the display 106, the projected image 112 in some respects mimicsa more traditional second display. This optional stand-alone featureallows the user to remove the unit 110, and thus the image projector108, from being coupled to the handheld controller 104, and have theimage projector 108 continue to project an image. Such removal may bedesirable if, for example, the user intends to subject the handheldcontroller 104 to many rapid motions but prefers the projected image tobe stable. Whenever the user chooses, the unit 110 can be docked backonto the handheld controller 104 so that the user once again can havemore control over the direction of the image projection.

Thus, in some embodiments, the unit 110 includes an audio/videostreaming capability through a wireless connection such as WiFi with theprocessor based apparatus 102, which may comprise a game console,entertainment system, PC, etc. In some embodiments, the unit 110includes a docking capability with a handheld controller. In someembodiments, the unit 110 can also operate as a stand-alone unit so thatthe user can, for example, undock it from the handheld controller andplace the unit 110 at a certain location.

Whether the unit 110 is docked to the handheld controller 104, or theunit 110 is undocked, or the image projector 108 is built into thehandheld controller 104, the image projector 108 may project any type ofimage(s). For example, the image projector 108 may project still images,moving images, video, or any other type of pictures or images. The imageprojector 108 may project images or video onto any surface. It isbelieved that projecting onto a flat surface will provide good results,but the projection may be onto any surface. Furthermore, as mentionedabove and as will be discussed more below, the image projector 108 mayproject images and/or video into the air to form a hologram.

In some embodiments, the processor based apparatus 102 will initiallydetect or determine whether or not an image projector is present, i.e.whether or not an image projector is coupled to, in communication with,and/or otherwise available to the system. If the processor basedapparatus 102 detects that an image projector is present, then in someembodiments the processor based apparatus 102 will generate and sendadditional content for projection. For example, additional scenes,characters, menus, or other images and/or video may be generated thatare specifically intended for projection by the image projector.

Thus, in some embodiments, the image, images, and/or video that areprojected from the image projector 108 are different than the image,images, and/or video being simultaneously displayed on the display 106.That is, the secondary images and/or video projected from the imageprojector 108 may or may not be the same images and/or video that aredisplayed on the display 106 at the same time or during the same timeperiod. In some embodiments, a totally different image may be used forthe projected image than what is displayed on the main display. In someembodiments, the images and/or video that are projected from the imageprojector 108 are different than the images and/or video beingsimultaneously displayed on the display 106, but the images and/or videothat are projected from the image projector 108 are synchronized withthe images and/or video being simultaneously displayed on the display106. Thus, in some embodiments, the images and/or video that areprojected from the image projector 108 are different, but they aresynchronized with the computer simulation and the images and/or videobeing simultaneously displayed on the display 106. As such, thisprovides the user with a different environment/image in which to playvideo games or other computer simulations.

Reference will now be made to FIGS. 4A, 4B, and 4C to discuss exampleuses of the image projector 108. As mentioned above, in some embodimentsthe image projector 108 is built into the handheld controller 104instead of using the unit 110. As such, the image projector 108 isillustrated as being housed in the “Unit 110 (or Handheld Controller104)” since the image projector 108 may be housed in either the unit 110or the handheld controller 104 in accordance with various embodiments ofthe present invention.

Referring specifically to FIG. 4A, a field map 130 may be projected bythe image projector 108 while the user plays a first person shooter gamethat is displayed on the display 106. That is, the shooting is displayedon the display 106 but is not projected by the image projector 108, andthe field map 130 is projected by the image projector 108 but is notdisplayed on the display 106. Thus, if the user needs to open a map, themap will be projected on the floor, wall, or some other surface insteadof being displayed on the main display 106. As such, the image projector108 adds a type of secondary display.

FIG. 4B illustrates another example. Namely, an item 132, such as aweapon like a sword or gun, may be projected by the image projector 108but not displayed on the display 106. This feature may be used, forexample, to allow the user to scroll through a list of items, such asweapons, the images of which are projected by the image projector 108while something else related (or even unrelated) to the game orsimulation is displayed on the display 106. Thus, instead of going tothe play menu to pop up options/menus/items on the same display 106, theoptions/menus/items can be shown separately from the display 106 byprojecting them onto the floor, wall, ceiling, etc., or even as ahologram, with the image projector 108.

FIG. 4C illustrates another example use for the image projector 108.Specifically, in some embodiments, the user may hear someone call him orher in the game, and then a face 134 pops up that is projected by theimage projector 108. The face 134 may, for example, be that of acharacter in the game or simulation, or a team member or friend, orsomebody else.

As mentioned above, in some embodiments the unit 110 also includes audiospeakers or an audio output jack so that sounds can be heard by the userwhile viewing the projected image. This allows audio to be streamed tothe unit 110 so that, for example, the face 134 can have a voice andtalk to the user, perhaps to give instructions, plan strategy, etc.Thus, in some embodiments the secondary images and/or video projectedfrom the image projector 108 also includes corresponding or associatedsecondary audio that is streamed to the unit 110 and played throughspeakers in the unit 110 or earbuds or headphones worn by the user. Insome embodiments, the secondary audio is considered to be associatedwith the secondary images and/or video projected from the imageprojector 108 when the audio corresponds to, goes along with, or belongswith the images or video, such as a voice that is intended to belong tothe person depicted in the images or video. As with the secondaryprojected images or video, the secondary audio may or may not be thesame audio that is played with the display 106. In some embodiments, thesecondary audio is not the same audio that is simultaneously played withthe display 106.

In some embodiments, the unit 110 also includes a microphone. Themicrophone allows the unit 110 to also detect sounds, such as a user'svoice, to send back to the processor based apparatus 102. This featureallows the user to talk to the projected image. For example, when theprojected face 134 talks to the user, the user can talk back and carryon a conversation with the face 134.

Thus, in some embodiments the unit 110 is capable of projecting imagesand/or video on any surface, or forming a hologram in the air, and alsoplaying sound that is associated with the images, video, or hologram. Assuch, audio can accompany the image or video projections. For example,the options/menus/items mentioned above can be projected by imageprojector 108 with associated audio.

In some embodiments, the image projector 108 projects images and/orvideo when the user uses the handheld controller 104 to make gesturalcommands. Specifically, if the handheld controller 104 is a motionsensing controller that is capable of sensing gestures, the computersimulation may generate images and/or video and send them to the imageprojector 108 so that the images and/or video are projected during oneor more of the gestural commands. This way, the image projector 108 addssomething extra to the gesture movement while the user is moving thehandheld controller 104 around to make the gestural commands. Forexample, in some embodiments, the image projector 108 can project ashooting star or a rainbow while the controller 104 is moving during thegesture movement to make the command more exciting and entertaining. Thestate of the computer simulation can use this feature when it knowsthere will be gestural commands. Some computer simulations, such as somevideo games, do not use gestural commands. Thus, the use of gesturalcommands will depend on the particular computer simulation or gametitle. If there will be no gestural commands, the computer simulationwill know that the handheld controller 104 will be relatively still andcan use the image projector 108 to project items such as menus, maps,etc.

In some embodiments, the image projector 108 can be mechanically movedto change the angle of projection. For example, in some embodiments, theimage projector 108 can be mechanically moved up/down and/or left/rightand/or swivel to change the angle of projection. As such, in someembodiments the image projector 108 is coupled to the handheldcontroller 104 in a manner that allows an angle of projection withrespect to the handheld controller 104 to be adjusted. In someembodiments, such coupling is accomplished by the image projector 108being built into the handheld controller 104 and being capable of havingits angle of projection adjusted. In some embodiments, such coupling isaccomplished by the image projector 108 being housed in the unit 110,which can be docked and/or attached to the handheld controller 104, andthe image projector 108 being capable of having its angle of projectionadjusted.

FIGS. 5A, 5B, and 5C illustrate examples of some embodiments of suchcoupling and the adjustment of the angle of projection of the imageprojector 108. Specifically, the image projector 108 is attached to theunit 110, or alternatively, the image projector 108 is attached directlyto the handheld controller 104. With the figures being side views, FIG.5A illustrates the image projector 108 being moved up, FIG. 5Billustrates the image projector 108 being moved so that it projectsparallel to the floor, and FIG. 5C illustrates the image projector 108being moved down. Thus, the angle of projection of the image projector108 is rotated from being directed up, to being directed level, to beingdirected down. The angle of projection changes with respect to thehandheld controller 104 whether the image projector 108 is attacheddirectly to the handheld controller 104, or to the unit 110 since theunit 110 may be releasably docked or attached to the handled controller104. In some embodiments, the image projector 108 swivels in order tochange the angle of projection.

In some embodiments, the angle of projection of the image projector 108is automatically adjusted according to a state of the computersimulation. For example, if the computer simulation comprises a videogame, and the image projector 108 is projecting the image of a characterin the game, the game state may at times send one or more signals orcommands to automatically adjust and/or rotate the angle of projectionof the image projector 108 in order to move the character or otherprojected image. As such, the game state can control the movement androtation of the image projector 108. In some embodiments, miniaturizedmotors or other movement means are employed to automatically move theimage projector 108 to change or adjust the angle of projection.

As mentioned several times above, in some embodiments the imageprojector 108 projects holographic images. Specifically, a hologram maybe formed based on the one or more images generated by the computersimulation. The image projector 108 projects the images and/or videointo the air to form the hologram. In some embodiments, a hologram isformed by projecting the images and/or video generated by the computersimulation in the direction of one or more clear panels or other type ofpanels that are appropriate for the creation of holograms. That is, theimages and/or video are directed at, into, or through one or more clearpanels or other type of panels that are appropriate for the creation ofholograms. In some embodiments, the one or more clear panels maycomprise acrylic panels.

In some embodiments, the clear panel may be coupled to the handheldcontroller. That is, in some embodiments, a small clear panel is coupledto the handheld controller so that it can project images and/or videoabove, beside, or below the handheld controller. FIGS. 6A and 6Billustrate examples of how a clear panel 140 may be coupled to thehandheld controller. Specifically, the clear panel 140 may be attachedto the unit 110, or alternatively, the clear panel 140 may be attacheddirectly to the handheld controller 104. In some embodiments, the clearpanel 140 may comprise an acrylic panel. In FIG. 6A the clear panel 140is attached to a side of the unit 110 or controller 104, which minimizesa combined length of the assembly. The image projector 108 is rotated sothat it projects into the clear panel 140. In FIG. 6B the clear panel140 is attached to an end of the unit 110 or controller 104, whichincreases the combined length of the assembly but minimizes its width.The image projector 108 is rotated slightly so that it projects at theclear panel 140 at an angle.

In some embodiments, the angle of the clear panel 140 is adjustable ortunable. That is, the angle of the clear panel 140 with respect to theunit 110 or handheld controller 104 is adjustable or tunable. In someembodiments, this adjustability or tunability may be accomplished byattaching the clear panel 140 to the unit 110 or handheld controller 104with a hinging, pivot, or axis mechanism 142.

In some embodiments, the angle of the clear panel is automaticallyadjusted according to a state of the computer simulation. For example,as part of the process of forming a hologram, the computer simulationmay automatically adjust or tune the angle of the clear panel 140, aswell as the angle of projection of the image projector 108, in order tocreate the proper conditions for forming a hologram. As another example,if the computer simulation comprises a video game, and the imageprojector 108 is using the clear panel 140 to project a hologram of acharacter in the game, the game state may automatically adjust or tunethe angle of the clear panel 140 and/or the angle of projection of theimage projector 108 in order to create the hologram, maintain thehologram, move the hologram, etc. In some embodiments, miniaturizedmotors or other movement means are employed or associated with thehinging, pivot, or axis mechanism 142 to automatically move the clearpanel 140 to change, adjust, or tune its angle.

In some embodiments, a hologram is formed by projecting the imagesand/or video generated by the computer simulation in a direction ofsmall or tiny particles suspended in the air. By way of example, thesmall or tiny particles suspended in the air may comprise fog, vapor,dust, or any other material that can be suspended in air and have lightprojected thereon. For example, the vapor produced by a device such asan e-cigarette is believed to be a material that can be suspended in airand have light projected thereon.

FIG. 7 illustrates an example of using small particles suspended in airto create a hologram. Specifically, a fog machine 150 creates fog 152.The image projector 108 projects images and/or video in a direction of,onto, into, or through the fog 152. The projection of light onto thematerial that comprises the fog 152 creates a hologram 154, which in theillustrated example is a game character.

In some embodiments, the small particles suspended in the air areautomatically established according to a state of the computersimulation. For example, as part of the process of forming a hologram,the computer simulation may automatically control the fog machine 150,or other particle producing apparatus, as well as the angle ofprojection of the image projector 108, in order to create the properconditions for forming a hologram. As another example, if the computersimulation comprises a video game, and the game state calls for thecreation of a holographic character 154, the game state mayautomatically control the fog machine 150, or other particle producingapparatus, as well as the angle of projection of the image projector108, in order to create, maintain, or move a hologram depicting thecharacter. That is, the game state will control the fog machine 150 torelease or establish fog 152 in the air, and then the game state willcontrol the image projector 108 to project images or video onto the fog152 to create the hologram 154.

Referring to FIG. 8, there is illustrated an example configuration ofhardware and circuity components that may be used in some embodiments inthe unit 110 to support the image projector 108. As illustrated, thecomponents include the image projector 108 (which may also be referredto as an optical block), a control circuit block 160, a wireless block162, an audio block 164, an interface block 166, an external interfaceblock 178, and a battery block 168.

As mentioned above, in some embodiments the image projector 108 maycomprise any type of image projector, such as for example, but notlimited to, a pico projector, handheld projector, or mobile projector.In some embodiments, the image projector 108 may comprise a microprojector module. Furthermore, in some embodiments the image projector108 may employ any imager technology, such as for example, but notlimited to, Digital Light Processing (DLP), Liquid Crystal on Silicon(LCoS), and laser.

The image projector 108 is coupled to the control circuit block 160. Insome embodiments, the control circuit block 160 may comprise amicroprocessor and a CODEC (coder-decoder). In some embodiments, thecontrol circuit block 160 is configured for implementing one or more ofthe features, methods, and techniques described herein.

The wireless block 162 is also coupled to the control circuit block 160.The wireless block 162 is used for receiving data from, and sending datato, the host, i.e. the processor based apparatus 102. For example, thewireless block 162 may be used for receiving one or more images and/orvideo generated by the computer simulation from the processor basedapparatus 102. The wireless block 162 may also be used for transmittingdata, such audio data for the user's voice, to the processor basedapparatus 102. As mentioned above, in some embodiments the wirelessblock 162 uses WiFi or another wireless connection technique to senddata between the processor based apparatus 102 and the unit 110.

The audio block 164 is also coupled to the control circuit block 160. Asmentioned above, in some embodiments the unit 110 includes one or moreaudio speakers 170 and/or an audio output jack 172 (for connection toexternal speakers) so that sounds can be heard by the user while viewingthe images or video projected by the image projector 108. In someembodiments, the speakers 170 and/or output jack 172 may be configuredfor stereo sound. The one or more audio signals received from theprocessor based apparatus 102 may be associated with the projectedimages or video. Furthermore, as also mentioned above, in someembodiments the unit 110 also includes a microphone 174. The microphone174 allows the unit 110 to detect sounds, such as a user's voice, tosend to the processor based apparatus 102 via the wireless block 162. Insome embodiments, voice recognition software or techniques may beemployed by the audio block 164 and/or the processor based apparatus 102for recognizing or translating voice received by the microphone 174.

The interface block 166, which is used in some embodiments, is alsocoupled to the control circuit block 160. The interface block 166 isused for interfacing with the handheld controller 104 when the unit 110is releasably docked with the handheld controller 104. The interfaceblock 166 couples to a corresponding interface 176, such as an expansioninterface, in the handheld controller 104. Thus, the interface block 166provides for communication between the unit 110 and the handheldcontroller 104. In some embodiments, the interface block 166 allows theunit 110 to use one or more resources located in the handheld controller104, such as for example, power from the handheld controller 104, one ormore audio speakers that might be in the handheld controller 104, amicrophone that might be in the handheld controller 104, control signalsgenerated by the handheld controller 104, etc. For example, somehandheld controllers are provided with audio signals so that users canplug earphones or headphones into the handheld controller. In someembodiments, the interface block 166 and corresponding interface 176allows the unit 110 to obtain the audio signals from the handheldcontroller 104. In such a scenario, in some embodiments, the unit 110does not need to obtain the audio signals from the processor basedapparatus 102.

The battery block 168, which is optional and is used in someembodiments, is also coupled to the control circuit block 160.Specifically, in some embodiments, power is provided to the unit 110 bythe handheld controller 104 when the unit 110 is docked with thehandheld controller 104. And when the unit 110 is undocked from thehandheld controller 104 and in stand-alone configuration, power can besupplied to the unit 110 from a wall plug. Thus, in these scenarios thebattery block 168 may not be needed and can be eliminated. But if thehandheld controller 104 cannot supply enough power to the unit 110 whenthe unit 110 is docked with the handheld controller 104, the batteryblock 168 may be needed. Similarly, the battery block 168 may be neededfor the stand-alone configuration if a wall plug is not available. Insome embodiments, the battery block 168 comprises rechargeable batterytechnology through the external interface block 178.

The external interface block 178, which is used in some embodiments, isalso coupled to the control circuit block 160 and the battery block 168.The external interface block 178 is used for interfacing with theprocessor based apparatus 102 when the wireless environment is notavailable and/or the wireless transceiver 162 is in sleep mode, or whenthe unit 110 is powered from an external power source. The externalinterface block 178 can provide for communication between the unit 110and the processor based apparatus 102. In some embodiments, the externalinterface block 178 allows the unit 110 to obtain one or more resourcesfrom the processor based apparatus 102, such as for example, power fromthe processor based apparatus 102, audio signals generated by theprocessor based apparatus 102, video signals generated by the processorbased apparatus 102, and control signals generated by the processorbased apparatus 102. In some embodiments, the external interface block178 allows the processor based apparatus 102 to use a microphone signalin the unit 110.

As mentioned above, in some embodiments the image projector 108 is builtinto the handheld controller 104. As such, in some embodiments the unit110 is not needed and many of the components illustrated in FIG. 8 areinstead housed in the handheld controller 104 along with the usualcomponents that make up the handheld controller 104. FIG. 9 illustratesan example configuration of hardware and circuity components that may beused in some embodiments in the handheld controller 104 to support theimage projector 108. Similar to the unit 110, the components include theimage projector 108 (which may also be referred to as an optical block),a control circuit block 160, a wireless block 162, an external interfaceblock 178, and an audio block 164, which operate similar to as describedabove. But unlike the unit 110, an interface block 166 for interfacingwith a handheld controller is not needed because the components arebuilt into the handheld controller 104. Furthermore, a battery block 168may not be needed if enough power is available in the handheldcontroller 104. However, if there is not enough power available in thehandheld controller 104, then a battery block 168 can certainly beincluded.

Thus, FIG. 9 illustrates an example of the image projector 108 beingcoupled to the handheld controller 104 by being built into the handheldcontroller 104. That is, FIG. 9 illustrates an example of the imageprojector 108 being included and/or mounted in the housing of thehandheld controller 104.

An example of the operational flow of the system 100 (FIGS. 1-3) forsome embodiments is illustrated by the method 200 shown in FIG. 10. Itshould be understood that not all of the steps of the method 200 arerequired in some embodiments. That is, some embodiments do not includeall of the steps of the method 200. Indeed, some of the steps of themethod 200 are optional. In some embodiments, one or more of the stepsof the method 200 may be executed by a processor based apparatus orsystem, such as the processor based apparatus 102 described above, whichin some embodiments may comprise a game console, entertainment system,handheld device, mobile device, personal computer, workstation, etc.

The method 200 begins in step 202 in which a computer simulation is run.The computer simulation is of a type that is controlled at least in partby a handheld controller, such as for example a game controller, gamepad, wand, mobile device, handheld device, or any of the other types ofhandheld controllers mentioned herein. In some embodiments, the computersimulation may comprise a computer game, video game, or any other typeof computer simulation.

In step 204, images generated by the computer simulation are sent to adisplay. The display may comprise any type of display, such as amonitor, TV, or any other type of display.

In step 206, the presence of an image projector is detected. Forexample, in some embodiments, a processor based apparatus will detect ordetermine whether or not an image projector is coupled to, incommunication with, and/or otherwise available to the system.

The result of step 206 is determined by decision block 208. If theprocessor based apparatus detects that an image projector is present,then the answer in decision block 208 is “yes”, which means in someembodiments the processor based apparatus will proceed to generateand/or send additional content for projection. As such, the process flowcontinues on to step 210 where the additional content is generated.

On the other hand, if the processor based apparatus does not detect thatan image projector is present, then the answer in decision block 208 is“no”, and the process flow returns to step 202. Thus, if the system doesnot have access to an image projector, then in some embodiments theprocessor based apparatus will only generate and/or send content for themain display.

If an image projector is present, then in step 210 at least one imagefor projection by the image projector is generated by the computersimulation. For example, additional scenes, characters, menus, or otherimages and/or video may be generated that are specifically intended forprojection by the image projector. As another example, in a video game,if the game state detects the presence of the image projector, then theprocessor based apparatus will generate the face of an enemy character,teammate, boss, maps, and/or other items for the image projector toproject. These items can be projected on the wall, the floor, or as ahologram in the air. In some embodiments, one or more images, movingimages, video, and/or holograms are generated. In some embodiments, thegenerated images and/or video are different than the images and/or videothat is being simultaneously displayed on the display. Thus, in someembodiments, when an image projector is present and/or available to thesystem, then additional content is generated for projection by the imageprojector.

In step 212, the at least one image for projection generated by thecomputer simulation is sent to the image projector. In some embodiments,the one or more images and/or video are sent to the image projector viawireless transmission. In some embodiments, the image projector iscoupled to the handheld controller in any of the ways described hereinabove.

In step 214, the image projector projects the at least one image forprojection generated by the computer simulation. In some embodiments,the at least one image is projected onto a surface. In some embodiments,the at least one image is used to form a hologram.

In some embodiments, the methods, schemes, and techniques describedherein may be utilized, implemented and/or run on many different typesof processor based apparatuses or systems. For example, the methods,schemes, and techniques described herein may be utilized, implementedand/or run on game consoles, entertainment systems, portable devices,mobile devices, pad-like devices, computers, workstations, desktopcomputers, notebook computers, servers, etc. Furthermore, in someembodiments the methods, schemes, and techniques described herein may beutilized, implemented and/or run in online scenarios, networkedscenarios, over the Internet, etc.

Referring to FIG. 11, there is illustrated an example of a system 300that may be used for any such implementations. The system 300 includes aprocessor based apparatus or system 320 that may also be used for anysuch implementations. For example, in some embodiments, one or morecomponents of the system 300 or the processor based apparatus or system320 may be used for implementing any method, scheme, technique, system,or device mentioned above, such as for example any of theabove-mentioned game consoles, entertainment systems, portable devices,mobile devices, pad-like devices, computers, workstations, desktopcomputers, notebook computers, servers, etc. However, the use of thesystem 300, the processor based apparatus or system 320, or any portionthereof is certainly not required. By way of example, in someembodiments, the system 300, or the processor based apparatus or system320, or any portions thereof, may be used for implementing the system100 (FIGS. 1-3) or any portion or component of the system 100 describedabove. For example, in some embodiments, the processor based apparatusor system 320 may be used for implementing the processor based apparatus102. In some embodiments, the display 312 may correspond to the display106, and the input device 316 may correspond to the handheld controller104.

By way of example, the processor based apparatus or system 320 (FIG. 11)may include, but is not required to include, a central processing unit(CPU) 302, a wireless and/or wired network interface 304, a randomaccess memory (RAM) 308, and a mass storage unit 310, such as a diskdrive. The system 320 may be coupled to, or integrated with, any of theother components described herein, such as a display 312 and/or an inputdevice 316. In some embodiments, the system 320 comprises an example ofa processor based apparatus or system. In some embodiments, such aprocessor based apparatus or system may also be considered to includethe display 312 and/or the input device 316. The CPU 302 may be used toexecute or assist in executing the steps of the methods, schemes, andtechniques described herein, and various program content, images, video,menus, menu screens, interfaces, graphical user interfaces (GUIs),windows, tables, graphics, avatars, characters, players, video games,simulations, virtual worlds, etc., may be rendered on the display 312.

In some embodiments, the wireless and/or wired network interface 304 maybe used to provide any necessary functionality for implementing any ofthe above-mentioned wireless streaming, wireless connections, WiFi, etc.For example, in some embodiments, the wireless and/or wired networkinterface 304 may be used to provide for the wireless streaming ofaudio, images and/or video between the processor based apparatus 102(FIGS. 1-3) and the unit 110, handled controller 104, and/or imageprojector 108. For example, in some embodiments, the wireless and/orwired network interface 304 may be used for wireless communications withthe wireless block 162 (FIGS. 8-9). In some embodiments, the wirelessand/or wired network interface 304 provides any necessary functionalityfor interfacing the system 320 with a network, such as a local areanetwork, the Internet, an intranet, a wide area network, or any othernetwork.

The input device 316 may comprise any type of input device or inputtechnique or method. For example, the input device 316 may comprise ahandheld controller, game controller, game pad, wand, motion sensingcontroller, joystick, keyboard, mouse, touch pad, trackball, camera,sensor, or other input devices and/or input techniques. The input device316 may be wireless or wired, e.g. it may be wirelessly coupled to thesystem 320 or comprise a wired connection. In some embodiments, theinput device 316 may comprise means or sensors for sensing and/ortracking the movements and/or motions of a user and/or an objectcontrolled by a user. The display 312 may comprise any type of displayor display device or apparatus, such as for example, but not limited to,a monitor, TV, etc.

The mass storage unit 310 may include or comprise any type of computerreadable storage or recording medium or media. The computer readablestorage or recording medium or media may be fixed in the mass storageunit 310, or the mass storage unit 310 may optionally include removablestorage media 314, such as a digital video disk (DVD), Blu-ray disc,compact disk (CD), USB storage device, floppy disk, or other media. Byway of example, the mass storage unit 310 may comprise a disk drive, ahard disk drive, flash memory device, USB storage device, Blu-ray discdrive, DVD drive, CD drive, floppy disk drive, etc. The mass storageunit 310 or removable storage media 314 may be used for storing code ormacros that implement the methods and techniques described herein.

Thus, removable storage media 314 may optionally be used with the massstorage unit 310, which may be used for storing program or computer codethat implements the methods and techniques described herein, such asprogram code for running the above-described methods, schemes, andtechniques. However, any of the storage devices, such as the RAM 308 ormass storage unit 310, may be used for storing such code. For example,any of such storage devices may serve as a tangible non-transitorycomputer readable storage medium for storing or embodying one or morecomputer programs or software applications for causing a game console,entertainment system, mobile device, system, computer, workstation,server, client, or other processor based apparatus or system to executeor perform the steps of any of the methods, code, schemes, and/ortechniques described herein. Furthermore, any of the storage devices,such as the RAM 308, mass storage unit 310, or removable storage media314 may be used for storing any needed database(s).

Thus, as mentioned above, in some embodiments, the processor basedapparatus or system 320 may be used for implementing the processor basedapparatus 102 (FIGS. 1-3). In some embodiments, the display 312 maycorrespond to the display 106, and the input device 316 may correspondto the handheld controller 104.

In some embodiments, one or more of the embodiments, methods,approaches, schemes, and/or techniques described above may beimplemented in one or more computer programs or software applicationsexecutable by a processor based apparatus or system. By way of example,such processor based system may comprise the processor based apparatusor system 320 (FIG. 11), or an entertainment system, game console,mobile device, computer, workstation, desktop computer, notebookcomputer, server, graphics workstation, client, portable device,pad-like device, etc. Such computer program(s) or software may be usedfor executing various steps and/or features of the above-describedmethods, schemes, and/or techniques. That is, the computer program(s) orsoftware may be adapted or configured to cause or configure a processorbased apparatus or system to execute and achieve the functions describedherein. For example, such computer program(s) or software may be usedfor implementing any embodiment of the above-described methods, steps,techniques, schemes, or features. As another example, such computerprogram(s) or software may be used for implementing any type of tool orsimilar utility that uses any one or more of the above describedembodiments, methods, approaches, schemes, and/or techniques. In someembodiments, one or more such computer programs or software may comprisea tool, utility, application, computer simulation, computer game, videogame, role-playing game (RPG), other computer simulation, or systemsoftware such as an operating system, BIOS, macro, or other utility. Insome embodiments, program code macros, modules, loops, subroutines,calls, etc., within or without the computer program(s) may be used forexecuting various steps and/or features of the above-described methods,schemes and/or techniques. In some embodiments, such computer program(s)or software may be stored or embodied in a non-transitory computerreadable storage or recording medium or media, such as any of thetangible computer readable storage or recording medium or mediadescribed above. In some embodiments, such computer program(s) orsoftware may be stored or embodied in transitory computer readablestorage or recording medium or media, such as in one or more transitoryforms of signal transmission (for example, a propagating electrical orelectromagnetic signal).

Therefore, in some embodiments the present invention provides a computerprogram product comprising a medium for embodying a computer program forinput to a computer and a computer program embodied in the medium forcausing the computer to perform or execute steps comprising any one ormore of the steps involved in any one or more of the embodiments,methods, approaches, schemes, and/or techniques described herein. Forexample, in some embodiments the present invention provides one or morenon-transitory computer readable storage mediums storing one or morecomputer programs adapted or configured to cause a processor basedapparatus or system to execute steps comprising: running a computersimulation that is controlled at least in part by a handheld controller;sending images generated by the computer simulation to a display; andsending at least one image generated by the computer simulation to animage projector coupled to the handheld controller. As another example,in some embodiments the present invention provides one or morenon-transitory computer readable storage mediums storing one or morecomputer programs adapted or configured to cause a processor basedapparatus or system to execute steps comprising: running a computersimulation that is controlled at least in part by a handheld controller;sending images generated by the computer simulation to a display; andsending at least one image generated by the computer simulation to animage projector that is housed in a unit configured to be releasablydocked to the handheld controller.

While the invention herein disclosed has been described by means ofspecific embodiments and applications thereof, numerous modificationsand variations could be made thereto by those skilled in the art withoutdeparting from the scope of the invention set forth in the claims.

What is claimed is:
 1. A method, comprising: running, with a processorbased apparatus, a computer simulation that is controlled at least inpart by a handheld controller; displaying images generated by thecomputer simulation on a display; and projecting at least one imagegenerated by the computer simulation from an image projector coupled tothe handheld controller.
 2. The method of claim 1, wherein the at leastone image projected from the image projector is different than theimages being simultaneously displayed on the display.
 3. The method ofclaim 1, further comprising: wirelessly streaming the at least one imagegenerated by the computer simulation to a location of the imageprojector coupled to the handheld controller before the at least oneimage generated by the computer simulation is projected from the imageprojector.
 4. The method of claim 1, further comprising: providing oneor more audio signals to a location of the image projector coupled tothe handheld controller, wherein the one or more audio signals aregenerated by the computer simulation and are associated with the atleast one image projected from the image projector.
 5. The method ofclaim 4, wherein the providing one or more audio signals to a locationof the image projector coupled to the handheld controller comprises:wirelessly streaming the one or more audio signals to a location of theimage projector coupled to the handheld controller.
 6. The method ofclaim 1, further comprising: automatically adjusting an angle ofprojection of the image projector according to a state of the computersimulation.
 7. The method of claim 1, wherein the projecting at leastone image generated by the computer simulation from an image projectorcoupled to the handheld controller comprises: forming a hologram basedon the at least one image generated by the computer simulation.
 8. Themethod of claim 1, wherein the projecting at least one image generatedby the computer simulation from an image projector coupled to thehandheld controller comprises: projecting the at least one imagegenerated by the computer simulation in a direction of a clear panel. 9.The method of claim 8, wherein the clear panel is coupled to thehandheld controller.
 10. The method of claim 8, further comprising:automatically adjusting an angle of the clear panel according to a stateof the computer simulation.
 11. The method of claim 1, wherein theprojecting at least one image generated by the computer simulation froman image projector coupled to the handheld controller comprises:projecting the at least one image generated by the computer simulationin a direction of small particles suspended in the air.
 12. The methodof claim 11, wherein the small particles suspended in the air comprisefog.
 13. The method of claim 11, further comprising: automaticallyestablishing the small particles suspended in the air according to astate of the computer simulation.
 14. A system, comprising: a display; ahandheld controller; an image projector coupled to the handheldcontroller; and a processor based apparatus in communication with thedisplay, the handheld controller, and the image projector; wherein theprocessor based apparatus is configured to execute steps comprising:running a computer simulation that is controlled at least in part by thehandheld controller; sending images generated by the computer simulationto the display; and sending at least one image generated by the computersimulation to the image projector.
 15. The system of claim 14, whereinthe at least one image generated by the computer simulation that is sentto the image projector is different than the images generated by thecomputer simulation being simultaneously sent to the display.
 16. Thesystem of claim 14, wherein the sending at least one image generated bythe computer simulation to the image projector comprises: wirelesslystreaming the at least one image generated by the computer simulation tothe image projector.
 17. The system of claim 14, wherein the processorbased apparatus is further configured to execute steps comprising:providing one or more audio signals to a location of the image projectorcoupled to the handheld controller, wherein the one or more audiosignals are generated by the computer simulation and are associated withthe at least one image generated by the computer simulation that is sentto the image projector.
 18. The system of claim 17, wherein theproviding one or more audio signals to a location of the image projectorcoupled to the handheld controller comprises: wirelessly streaming theone or more audio signals to a location of the image projector coupledto the handheld controller.
 19. The system of claim 14, wherein theprocessor based apparatus is further configured to execute stepscomprising: automatically adjusting an angle of projection of the imageprojector with respect to the handheld controller according to a stateof the computer simulation.
 20. The system of claim 14, wherein theprocessor based apparatus is further configured to execute stepscomprising: controlling the image projector to form a hologram based onthe at least one image generated by the computer simulation.
 21. Thesystem of claim 14, wherein the processor based apparatus is furtherconfigured to execute steps comprising: controlling the image projectorto project the at least one image generated by the computer simulationin a direction of a clear panel.
 22. The system of claim 14, wherein theprocessor based apparatus is further configured to execute stepscomprising: controlling the image projector to project the at least oneimage generated by the computer simulation in a direction of smallparticles suspended in the air.
 23. A non-transitory computer readablestorage medium storing one or more computer programs configured to causea processor based system to execute steps comprising: running a computersimulation that is controlled at least in part by a handheld controller;sending images generated by the computer simulation to a display; andsending at least one image generated by the computer simulation to animage projector coupled to the handheld controller.
 24. Thenon-transitory computer readable storage medium of claim 23, wherein theone or more computer programs are further configured to cause theprocessor based system to execute steps comprising: detecting a presenceof the image projector.
 25. The non-transitory computer readable storagemedium of claim 24, wherein the one or more computer programs arefurther configured to cause the processor based system to execute stepscomprising: in response to detecting a presence of the image projector,the computer simulation generating the at least one image so that whensent to the image projector it will be different than images beingsimultaneously sent to the display.
 26. A non-transitory computerreadable storage medium storing one or more computer programs configuredto cause a processor based system to execute steps comprising: running acomputer simulation that is controlled at least in part by a handheldcontroller; sending images generated by the computer simulation to adisplay; and sending at least one image generated by the computersimulation to an image projector that is housed in a unit configured tobe releasably docked to the handheld controller.
 27. The non-transitorycomputer readable storage medium of claim 26, wherein the one or morecomputer programs are further configured to cause the processor basedsystem to execute steps comprising: detecting a presence of the imageprojector.
 28. The non-transitory computer readable storage medium ofclaim 27, wherein the one or more computer programs are furtherconfigured to cause the processor based system to execute stepscomprising: in response to detecting a presence of the image projector,the computer simulation generating the at least one image so that whensent to the image projector it will be different than images beingsimultaneously sent to the display.